Re: LF: Coherent BPSK on LF using EbNaut 
Von: Markus Vester <markusvester@aol.com> 
An: rsgb_lf_group <rsgb_lf_group@blacksheep.org> 
Datum: So, 27 Sept 2015 10:40 pm 

Thanks very much to Domenico IZ7SLZ for informing the group about our LF EbNaut-PSK activities! I am glad that this has raised some interest, and I'm sure that trying Paul's software is well worth the effort for anyone wishing to communicate with really weak signals, very near to the ultimate Shannon-Hartley limit. 

In this mail, I will try to provide some information regarding the receive procedure. As mentioned before, we have used SpecLab as a receiving frontend to generate an IQ wav file, which can be read by EbNaut-RX. Wolf has provided a dedicated riggered audio recording function, which is able to save decimated and precisely timestamped IQ data, starting at specified GPS-controlled times. 

As I am already continuously running a SpecLab instance for the opds decoder, I decided to go down a slightly different route, by reusing the exported ascii spectrum-data files for EbNaut as well. One advantage of this procedure is that the LO used in the downconversion for the spectrogram already includes samplerate tracking, which avoids the need for CPU-intensive samplerate interpolation. If GPS-1pps is used for reference, the absolute LO phase will even be recovered after a soundcard dropout. Timestamping is derived from NTP, which is slightly less accurate but perfectly adequate for the symbol durations down to 0.2 s that were  used in our tests.

So all one has to do is to select a MMDDhhmm.txt file from the spectrum\data\ directory which contains the received sequence, and convert it back to time domain using an inverse FFT. In an email to Domenico (pasted beneath), I briefly described the file conversion and the frequency-time alignment process used with EbNaut. An updated version of the utilities can be found at
http://df6nm.bplaced.net/VLF/fec_tests/df6nm_ebnaut_utilities.zip (184 kB).
The SpecLab config-file _opds32_150924.usr for opds has been slightly updated, hopefully making life a bit it easier for first-time opds and EbNaut users. The analyzed audio frequency band has been centered on 1500 Hz. You will still have to calibrate your samplerate and receiver frequency. Note that for opds and EbNaut postprocessing the number of exported FFT bins (262144) always needs to be half the FFT length (524288).

The inverse-FFT utility is ebnaut_ifft3.exe. Dragging and dropping a .txt data file over it's icon will let it produce an equally named .wav file which can then be read by ebnaut-rx software. Note that for correct time and frequency information in the wav header, the ifft configuration file sr.txt needs to contain your accurate soundcard samplerate, FFT decimation factor and length, and RX frequency. 

Shortly after starting ebnaut-rx and before it enters the decoding process, it writes the symbol amplitudes and phases to a logfile called rawsyms.txt. This can be used to assist the determination of the correct frequency and time offset. Clicking on the show_rawsyms3.exe utility will output a window with three different diagrams and two lines of text information. 

The white dots at the top are a time domain representation of the symbol phases. With a strong signal, these should ideally be gathered around two horizontal lines, 180 apart. For the example screenshot, the frequency offset had been made 1 mHz too negative, resulting in gradually increasing symbol phases.

The red spectrum is a Fourier transform of the symbol amplitudes. The Nyquist frequency range is identical to the symbol rate (ie +- 0.5 Hz for 1 second symbols), with a zoomed section on the right. The spectrom will show a peak if a test carrier has been sent, and the freq offset in EbNaut can then be adjusted iteratively to bring it to the center. However no red peak will be appear from a PSK data transmission.   

The blue spectrum is calculated from the squared complex amplitudes. As taking the square eliminates the signal polarity, this spectrum will also show a peak from a PSK modulated sequence. Unfortunately the squared spectrum fails to produce a peak for very weak signals where the SNR of the individual symbols becomes too low. If he peak is present, the frequency error (e.g. offset needs to be increased by 0.97 mHz) can be read in the text beneath.

Unfortunately there is no direct indication to guide you towards the correct bit timing. One way is to iteratively adjust the fractional time offset for least scatter of dots or highest blue peak. In the bottom line, an estimate of the time error magnitude (nominally 0 to 0.5) is given, which should be small if the correct symbol timing has been found. This is calculated by a normalized autocorrelation with a one-symbol shift, exploiting the property of the sequence that the likelyhood of a symbol transition should always be 50%, independent of a prior transition. Again, this method will not work if the SNR is low.
 
For testing purposes, you can try the data in
http://df6nm.bplaced.net/VLF/fec_tests/ebnaut_data_09200300.zip (750 kB).
If you carefully inspect the timestamp of the successful decode, you may note that in this case an extra one-second delay was required. Domenico confirmed that it was caused by a temporary GPS-derived clock error on the transmit PC. 


Hope this helps... so go ahead and have fun! 

Thanks to Paul and Wolf for their original ideas and invaluable software, and to Domenico for surmounting all obstacles to bring this to life on 137 kHz.

Best 73, 
Markus

-----Ursprngliche Mitteilung----- 
Von: IZ7SLZ <iz7slz.domenico@gmail.com>
 An: rsgb_lf_group <rsgb_lf_group@blacksheep.org>
 Verschickt: Mi, 23 Sept 2015 2:06 am
 Betreff: LF: Coherent BPSK on LF using EbNaut

Hello LF,

I am pleased to report a successful "Coherent BPSK" QSO between DF6NM and IZ7SLZ on LF using EbNaut program of Paul Nicholson.
QSO took place in the night between Friday 2015-09-18 and Saturday 2015-09-19 .

I think it was the first time that anyone tried this on 137 kHz !

Frequencies used were 137510.00 Hz (DF6NM) and 137502.12 (IZ7SLZ).

We exchange messages of 16 characters in coherent BPSK with UT synchronous symbols, using a symbol period of 1 second and 8K19A scheme coding.

Software used were the Windows' versions of EbNaut-TX and EbNaut-RX (tnx to P. Nicholson: your software is working without problems!).
Also Spectrum Lab (tnx Wolf !) was used to export FFT data, in a similar way of the Markus' OPDS setup.
Other programs,  written by Markus,  were used: one  to convert FFT exported data for the EbNaut entry and another to search for the best frequency and time offsets.

There was not much phase drift during the qso and in other test transmissions, since we reach a good frequency stability at both ends thanks to Rubidium Reference Oscillator (Markus side) and a good quality OCXO from my side.

You can see reported symbol / carrier EbN0 and also the messages exchanged,  in the attached screen-shots taken at both ends.

We hope now to joint other stations that want to challenge the timing and frequency accuracy, to have a QSO in LF close to the Shannon limit.

Markus DF6NM, will report soon all particulars and his comments of the experiment. Thanks in advance Markus!

73, Domenico (iz7slz)
_________________________________
Re: EbNaut - success again! 
Von: Markus Vester <markusvester@aol.com> 
An: iz7slz.domenico <iz7slz.domenico@gmail.com> 
Cc: paul <paul@abelian.netcom.co.uk> 
Datum: Mi, 19 Aug 2015 4:58 pm 

Hi Domenico,

here's some guidance towards setting up EbNaut for receiving, using your opds SpecLab instance. 

1. Generating a stable test transmission

For accurate transmissions, it would be better to avoid the SSB and audio carrier method as it is prone to soundcard glitches. You may rather want to generate the carrier directly at RF (eg from your AD9850 DDS), and apply the DBM modulation there. In your setup, you might need to make a small predriver amplifier to feed the PA.

2. Decoder dry test

First try a dry test of the software using the Sunday 23:14 data file. Download the conversion program
http://df6nm.bplaced.net/VLF/fec_tests/ebnaut_ifft2.zip
(including my calibration data file sr.txt) and the data
http://df6nm.bplaced.net/VLF/fec_tests/08162314.zip .
Open a command shell and type
 ebnaut_ifft2 08162314
which will output a couple of lines and generate a 08162314.wav file. Load this into ebnaut-rx, using parameters as shown in attached screenshot (start offsets 248.5s, freq offset 2.0369 Hz). This should reproduce the perfect decode.

3. Setting up the opds SpecLab instance

As you have successfully received opds no big changes are nessary. PC time needs to be accurate to 0.2 s using NTP or GPStime. I'd recommend to activate samplerate tracking, using either a tone or 1 pps signal.

4. Saving data files

If opds detector software is running in parallel, disable the file deletion in cleanup bat, so that the *.txt files will not be removed from the c:/spectrum/data directory, e.g. by outcommenting the second and third lines
 @copy headline.txt + DETECTED.TXT + detecold.txt c:\dropbox\public\detected.txt >nul
 ::@del c:\spectrum\data\*.txt 1>nul 2>nul
 ::@copy dirw.txt dirdone.txt >nul 
Note that changing cleanup.bat will take effect immediately, without having to restart opds.

A new data file will be created every 10 minutes, with the filename corresponding to minute of the middle of the aquisition period. Note that in the long run this may accumulate a large amount of data on your disk. You can reduce filesize by selecting the "use INTEGER numbers only" option in the "File - Export.. - Export of FFT results" menu. In that case please check the data file output, if it consists of many zeroes or single digit numbers, you'll need to increase the audio level. 

5. Opds calibration

You should then perform an accurate frequency calibration. Read the samplerate from the samplerate detector. The RF center frequency can best be checked using Loran lines. In the mediteranean area, lines from Soustons should be detectable in daytime at exact multiples of 50000/6731 = 7.428316743 Hz, eg. 137505.5712 Hz (see https://dl.dropboxusercontent.com/u/26404526/Loran_lines_in_LF_slots.txt ). Use the cursor to read their display frequency and adjust the Options - ..part 2.. - Radio Freq Offset entry according to the error. You can then then enter use the calibration  in opds.ini.

6. Converting to a wav file

Then you need to select a data file containing an ebnaut signal or a test carrier, and convert it to a wav file. To use the ifft program, you should first edit sr.txt and enter your RX calibration result. This file contains four numbers, which are the soundcard samplerate, the FFT decimation factor, the FFT length (usually 524288), and the center frequency. Copy the data file from spectrum/data and run ebnaut_ifft2 as before.

7. Searching frequency offset

Start Paul's ebnaut-rx software and open a wav file which contains a pure and reasonbly strong carrier. Enter the difference to the calibrated center frequency (positive for signals above center), and start a decode using 1 sec symbol duration for 1 Hz search range, and eg. 16 for message length. Within a couple of seconds, ebnaut will have preprocessed the audio and written a rawsyms.txt file. The decode attempt can then be aborted by pressing Stop. 

We now have to inspect rawsys.txt, which contains the complex baseband signal at one sample per symbol. We want to plot the contents of the fourth column showing the phase. I am using the attached MathCad script for this, but you might as well employ Excel, gnuplot, or any other software. Ideally the data points should be gathered along a horizontal line, indicating constant phase and zero frequency error. Usually the line will have a slope, and you will have to adjust the Freq offset field and repeat the procedure. Finally you should do a fine-adjust using a longer symbol duration like 10 seconds.

Unfortunately this is a lengthy procedure, and it becomes very difficult or impossible when the signal is weak. This is the reason why it's so important to have a stable and repeatable frequency. At LF, a simple TCXO usually has too much variation, and we really want a Rubidioum or GPS-controlled oscillator, or a very stable OCXO. 

8. Searching time offset

Now we can go about decoding a wav containing a real ebnaut sequence. The remaining task isto determine the time offset between the beginning of the file and the sequence start. Theoretically it can be obtained from the difference of the displayed file start (e.g. 22:55:52.610) and the known transmission (e.g. 23:00:00, in this case 4 minutes and 7.39 seconds, i.e. 247.39 seconds). Unfortunately, there usually seems to be a small extra offset (around 0 to 2 seconds) which is initially unknown. So we will need a little trial and error.

Now start ebnaut with the theoretical start offset, and look at rawsyms.txt again. If the signal is strong enough, the dots should now gathered around  dots around two lines, 180 degrees apart. Most likely there will a number of dots scattered elsewhere, due to a time error which leads to sampling during symbol transitions.

Then try to repeat the procedure, varying the start offset in steps of 0.2 symbol periods (eg. 0.2 sec), trying to minimize the number of scattered points between the lines. If the signal is well above the noise you will find a clear optimum. If the two lines are still sloping, adjust the frequency offset in small steps (+-0.0002 Hz).

Let Ebnaut run a couple of minutes. If you are lucky you will already be getting a decode. 

If no decode happens, the start time may still be wrong by an integer number of symbols periods. Then try to increase time offset in steps of one symbol duration. With one of these offsets, the signal should be decoded successfully. 

You can still play around with the time offset to reduce symbol errors and maximize EbN0. Take a note of the best start offset. Assuming that the opds spectrogram is running continuously, a new file will be written exactly every 10 minutes. Thus the start offset can be reused for the next transmission, e.g. one hour later. After a SpecLab restart, the file start times will have changed arbitrarily, but the empirical extra offset will likely still be valid. 


Maybe we can do a test from me to you on the weekend? However before going on air, I want to insert a simple RC shaping network before the modulator, to avoid splattering up Stefan's grabber and opds screen too much.

All the best,
Markus



_________________________________
_________________________________
Re: LF: Coherent BPSK on LF using EbNaut 
Von: Domenico IZ7SLZ <iz7slz.domenico@gmail.com> 
An: rsgb_lf_group <rsgb_lf_group@blacksheep.org> 
Datum: Fr, 25 Sept 2015 4:39 pm 

Hello Andy,  all

thanks for interesting at this test.

Yes, Spectrum Lab is the right program to manage the audio from receiver also in this case. The SL configuration used in LF_EbNaut experiment basically is the same of the DF6NM OPDS setup  downloadable at Markus' site:

http://df6nm.bplaced.net/opera/opds.zip

Markus, as usual, includes all informations necessary to do the proper changes according to the hardware at disposal.

In my setup, i used SL to do many things: calibrate the sample-rate,  manage the ptt at scheduled time,  generate a precise tone to drive my SSB exciter,  create some screenshots of the waterfall and - most important -  export, every 10 minutes,  the FFT raw data of the reception periods.

Markus will explain better than me this matter, but exporting FFT instead of .WAV file seems a good solution to avoid some issues encountered with the  .wav  recording function of SL.

He wrote a program (named ebnaut_ifft2.exe) able to convert FFT raw data  to wav file,  inserting the appropriate timestamp.

After this conversion,  Ebnaut-rx can be used, but it needs of some - very sensitive -  offset entry. For that,  you need to become skilled with the use of some other Markus' tools that plot the symbol phase before the full EbNaut-rx search.

But, if the clock  and the  frequency have very small drift in both TX and RX sides, than you can use the same measured parameters to catch the next transmitted messages. 

73, Domenico (iz7slz)
_________________________________
On 25 September 2015 at 13:23, Andy Talbot <andy.g4jnt@gmail.com> wrote:

EbAnut is a new one here - I hadn't heard of it.   After Googling I found the page and read up / downloaded the software.   Unfortunately, I see it is not standalone, and requires Spectrum lab to act as the audio "front end" to take in the Received audio, downconvert to a baseband I/Q and store that as a .WAV file.   So downloaded Spectrum lab and tying to wade through how to use that.

Could you perhaps give a short cut simple explanation of how to set up Spectrum lab to do that - it looks to be a very complex bit of software to drive.

the EbNaut Tx and Rx programmes, in contrast, look very simple to use and it would be goo dto get more people using sensitive coherent signalling such as this.

Some years ago I proposed GPS / UTC seconds locked coherent signalling on LF, but didn't have the expertise to write the software - and certainly not to implement strong FEC !

Andy  G4JNT
_________________________________
On 23 September 2015 at 01:04, IZ7SLZ <iz7slz.domenico@gmail.com> wrote:

Hello LF,

I am pleased to report a successful "Coherent BPSK" QSO between DF6NM and IZ7SLZ on LF using EbNaut program of Paul Nicholson.
QSO took place in the night between Friday 2015-09-18 and Saturday 2015-09-19 .

I think it was the first time that anyone tried this on 137 kHz !

Frequencies used were 137510.00 Hz (DF6NM) and 137502.12 (IZ7SLZ).

We exchange messages of 16 characters in coherent BPSK with UT synchronous symbols, using a symbol period of 1 second and 8K19A scheme coding.

Software used were the Windows' versions of EbNaut-TX and EbNaut-RX (tnx to P. Nicholson: your software is working without problems!).
Also Spectrum Lab (tnx Wolf !) was used to export FFT data, in a similar way of the Markus' OPDS setup.
Other programs,  written by Markus,  were used: one  to convert FFT exported data for the EbNaut entry and another to search for the best frequency and time offsets.

There was not much phase drift during the qso and in other test transmissions, since we reach a good frequency stability at both ends thanks to Rubidium Reference Oscillator (Markus side) and a good quality OCXO from my side.

You can see reported symbol / carrier EbN0 and also the messages exchanged,  in the attached screen-shots taken at both ends.

We hope now to joint other stations that want to challenge the timing and frequency accuracy, to have a QSO in LF close to the Shannon limit.

Markus DF6NM, will report soon all particulars and his comments of the experiment. Thanks in advance Markus!

73, Domenico (iz7slz)
_________________________________
